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CLAIMS 



We claim: 

\ 1. A method comprising: 

\identifying, in response to a search query, first multimedia objects having 
an associated keyword that matches a keyword in the search query and second 
multimedia \bjects that have content features similar to those of the first 
multimedia objeofc; 

presenting thesfirst and second multimedia objects to a user; 

monitoring feedback from the user as to which of the first and second 
multimedia objects are relevant to the search query; 

annotating one or mor^sof the multimedia objects, which are deemed 
relevant by the user, with the keyword. 

2. A method as recited in claim lMurther comprising: 

maintaining associations between the keywords and the multimedia objects, 
the associations being weighted to indicate how relevant the keywords are to the 
multimedia objects; and \ 

adjusting the weights of the associations based on the user's feedback. 

3. A method as recited in claim 2, wherein the aqjusting comprises 
increasing a weight of an association between the keyword and a particular 
multimedia object that is deemed relevant by the user. \ 
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♦ ♦ 

y 4. A method as recited in claim 2, wherein the adjusting comprises 
decreasing a weight of an association between the keyword and a particular 
multiniedia object that is deemed irrelevant by the user. 

5. >A method as recited in claim 4, further comprising removing the 
keyword from the particular multimedia object in an event that the weight is less 
than a threshold value. 

6. A method as recited in claim 1 5 further comprising training how the 
first and second multimedia objects are identified based on the user's feedback. 

7. A method as recited in claim 1, further comprising refining the search 
to identify additional multimedia objects that contain content features similar to 
those of the multimedia objects indicated by thevuser as being relevant. 

8. A method as recited in claim 1, wherein the multimedia objects 
comprise one of digital images, video objects, and audio objects. 

9. A computer readable medium having computer-executable 
instructions that, when executed on a processor, perform the methoa\as recited in 
claim 1. \ 
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, 10. A method comprising: 

\ iteratively retrieving multimedia objects from a database and monitoring 
feedback from a user as to whether the multimedia objects are relevant to a 
keyworckin a search query; and 

annotating the multimedia objects based on the user's feedback. 

11. A method as recited in claim 10, wherein the retrieving comprises 
using content-based information retrieval to retrieve the multimedia objects. 

12. A method as recited in claim 10, wherein the retrieving comprises 
using both content-based inforhaation retrieval and semantic-based information 
retrieval to retrieve the multimedia objects. 

13. A method as recited in claimNlO, wherein the monitoring comprises 
monitoring both feature-based relevance feedback and semantic-based relevance 
feedback. \ 

14. A method as recited in claim 10, wherein the annotating is hidden 
from the user. \ 

15. A method as recited in claim 10, wherein the annotatings^omprises: 
in an event that a particular multimedia object is deemed relevant by the 

user and is not yet annotated with the keyword, adding the keyword r& the 
particular multimedia object; and \ 
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\ in an event that the particular multimedia object is deemed relevant by the 
user and is already annotated with the keyword, strengthening an association 
between the keyword and the particular multimedia object. 

16. X method as recited in claim 10, wherein the annotating comprises: 
in an evenrthat a particular multimedia object is deemed irrelevant by the 

user and is already annbtated with the keyword, weakening an association between 
the keyword and the particular multimedia object. 

17. A computer readabk medium having computer-executable 
instructions that, when executed on a processor, perform the method as recited in 
claim 10. \ 

18. A method comprising: \ 

retrieving multimedia objects according to k content-based retrieval 
process; \ 

presenting the multimedia objects to a user; \ 

monitoring feedback from the user as to which of the multimedia objects 
are relevant; and \ 

annotating one or more of the multimedia objects based on theNuser's 
feedback. \ 
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\ 19. A method as recited in claim 18, wherein the monitoring comprises 
monitoring both feature-based relevance feedback and semantic-based relevance 
feedoack. 

20. \ A method as recited in claim 1 8, wherein the annotating is hidden 
from the user A 

21. A method as recited in claim 18, wherein the annotating comprises: 
in an event that^a particular multimedia object is deemed relevant by the 

user and not yet annotatedVith the keyword, adding the keyword to the particular 
multimedia object; and \ 

in an event that the particular multimedia object is deemed relevant by the 
user and is already annotated witnVthe keyword, strengthening an association 
between the keyword and the particular multimedia object. 

22. A method as recited in claim 1 8,\wherein the annotating comprises: 
in an event that a particular multimedia olsnect is deemed irrelevant by the 

user and is already annotated with the keyword, weakening an association between 
the keyword and the particular multimedia object. \ 

23. A method as recited in claim 18, wherein the annotating comprises: 
in an event that a particular multimedia object is deemed irrelevant by the 

user and is already annotated with the keyword, removing the keyword from the 
particular multimedia object. \ 
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v 24. A computer readable medium having computer-executable 
inductions that, when executed on a processor, perform the method as recited in 
claim \8. 

25. \ A method comprising: 

maintaining associations between keywords and multimedia objects, the 
associations being\veighted to indicate how relevant the keywords are to the 
multimedia objects; ancl 

retrieving a set ofN)ne or more multimedia objects for presentation to a 
user; \ 

monitoring feedback fromNthe user as to which of the multimedia objects 
are relevant; and \ 

adjusting the weights of the associations based on the user's feedback. 

26. A method as recited in claim 25,\vherein the retrieving comprises 
using content-based information retrieval to retrieve the multimedia objects. 

27. A method as recited in claim 25, wherein trie retrieving comprises 
using both content-based information retrieval and semantiVbased information 
retrieval to retrieve the multimedia objects. \ 

28. A method as recited in claim 25, wherein the monitoringScomprises 
capturing both feature-based relevance feedback and semantic-based refo/ance 
feedback. \ 
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\ 29. A method as recited in claim 25 , wherein the adjusting comprises 
increasing the weights of the associations between the keywords and the 
multimedia objects that are deemed relevant by the user. 

30. \ A method as recited in claim 25, wherein the adjusting comprises 
decreasing thk weights of the associations between the keywords and the 
multimedia objects that are deemed irrelevant by the user. 

31. A computer readable medium having computer-executable 
instructions that, when exectrted on a processor, perform the method as recited in 
claim 25. \ 

32. A system comprising: \ 

an information retrieval unit to retrieve multimedia objects from a database 
based on a search query; \ 

a relevance feedback unit to capture a user's feedback as to whether the 
multimedia objects are relevant to the search query; ahd 

an annotation unit to annotate the multimedia oknects based on the user's 
feedback. \ 

33. A system as recited in claim 32, wherein the searcnSquery comprises 
a keyword-based search query having one or more keywords. \ 
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\ 34. A system as recited in claim 32, wherein the search query comprises 
a cotatent-based search query having one or more content features. 



35. \ A system as recited in claim 32, wherein the information retrieval 
unit employs\ both content-based information retrieval and semantic-based 
information retrieVal. 

36. A system as recited in claim 32, wherein the information retrieval 
unit comprises: \ 

a query handler to handlesboth keyword-based queries having one or more 
search keywords and content-based queries having one or more content features of 
a multimedia object; and \ 

a feature and semantic matcher th. identify at least one of (1) first 
multimedia objects having keywords that match the search keywords from a 
keyword-based query, and (2) second multimedia objects having content features 
similar to the content features of a content-based query \ 

37. A system as recited in claim 32, wherein the relevance feedback unit 
employs both feature-based relevance feedback and semantic-hased relevance 
feedback. \ 

38. A system as recited in claim 32, wherein: \ 
the search query comprises a keyword-based search query having at least 

one keyword; and 
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in an event that a particular multimedia object is deemed relevant by the 
uW and is not yet annotated with the keyword, the annotation unit adds the 
keyword to the particular multimedia object. 



39\ A system as recited in claim 32, wherein: 

the search query comprises a keyword-based search query having at least 
one keyword; and 

in an event\hat a particular multimedia object is deemed relevant by the 
user and is already annotated with the keyword, the annotation unit strengthens an 
association between the keyword and the particular multimedia object. 

40. A system as recited in claim 32, wherein: 

the search query comprises^ keyword-based search query having at least 
one keyword; and \ 

in an event that a particular multimedia object is deemed irrelevant by the 
user and is already annotated with the keyword, weakening an association between 
the keyword and the particular multimedia object. 



41. A system as recited in claim 32, wherem: 

the search query comprises a keyword-based search query having at least 
one keyword; and \ 

in an event that a particular multimedia object is deeded irrelevant by the 
user and is already annotated with the keyword, removing the keyword from the 
particular multimedia object. \ 
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42>v An image retrieval system as recited in claim 32, wherein the 
relevance feedbai&^unit comprises a feedback analyzer to train the system based 
on the user's feedback. 



43. A user interface, comprising: 

a query interface to accept a search query for searching multimedia objects 
in a database system; I \ , 

a browser to permit a us^to browse the multimedia objects returned from 
the database system; and \/ ^ 

a feedback interface te* enable the user to indicate which multimedia objects 
are relevant to the searchWeries. 

44. A user interface as recited in claim 43 5 wherein the query interface 
is configured to allow entr>\ of both keyword-based queries with one or more 
keywords and content-based queries based on selection of an example multimedia 
object. \ 
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